Polynomial Regression
Core Concept
Polynomial regression extends linear regression by including powers (and optionally products) of the features in the design matrix. For a single feature (x), the model might be (y = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_d x^d); for multiple features, polynomial terms include (x_i^2), (x_i x_j), etc., up to a chosen degree. The model remains linear in the parameters—each term is a new “feature” with its own coefficient—so the same least-squares machinery (normal equation or gradient descent) applies. The result is a parametric curve or surface that can capture curvature and non-linear trends while staying interpretable and within the linear-algebra framework of regression.
Key Characteristics
- Linear in parameters, non-linear in inputs – Fitting is still minimizing sum of squared errors over a linear combination of basis functions; the “features” are (1, x, x^2, \ldots, x^d) (and cross-terms if multivariate). Coefficients are interpretable as weights on each basis term, though individual (\beta_j) are less directly interpretable than in pure linear regression.
- Degree and overfitting – Higher degree allows more curvature and better fit to training data but increases overfitting risk and sensitivity to outliers. Degree acts as a complexity knob; cross-validation or validation-set performance is used to choose degree or to regularize polynomial coefficients.
- Numerical stability – Powers of (x) can have very different scales (e.g. (x^{10}) vs (x)); centering and scaling features, or using orthogonal polynomials, improves conditioning of the design matrix and stability of the solution.
- Extrapolation risk – Polynomials tend to diverge sharply outside the range of training data; predictions beyond the observed (x) range are often unreliable. Use is best restricted to interpolation or accompanied by clear uncertainty statements.
- Basis expansion view – Polynomial regression is a special case of basis expansion; the same idea extends to splines, Fourier terms, or other bases that capture non-linearity while keeping the model linear in parameters.
Common Applications
- Trend and growth modeling – Capturing non-linear trends over time or over a single predictor (e.g. quadratic or cubic growth, saturation)
- Calibration and response curves – Modeling instrument response or dose–response as a smooth curve when theory suggests a simple polynomial form
- Economics and demand – Representing non-linear price–quantity or income–demand relationships with quadratic or higher-order terms
- Engineering and physics – Approximating relationships that are known to be smooth and possibly polynomial in a transformed variable
- Feature engineering for linear models – Adding (x^2), (x_i x_j), or other polynomial terms to a linear model to capture curvature and interaction without leaving the linear regression framework
- Baseline for non-linearity – Testing whether a simple polynomial improves over linear regression before turning to splines, trees, or neural networks